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ABSTRACT 



Disclosed is a computer audio interface having stereo 
and filtered sound effects to enable blind users to oper- 
ate a graphical user interface. Stereo balance and incre- 
mental filtering are used along separate axes to guide a 
blind or visually impaired user within an area of a 
graphical user interface, particularly the client area of a 
window. As the pointer approaches the left boundary of 
the client area, the sounds representing the client area 
come more and more exclusively from the left audio 
channel. Likewise, when approaching the right bound- 
ary, the sound shifts to the right channcL Additionally, 
as the pointer is moved toward the top of the window 
client area, the pitch of the sound increases in stepwise 
fashion. 

10 Claims, 2 Drawing Sheets 
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COMPUTER USER INTERFACE WITH STEREO 
AND FILTERED SOUND EFFECTS 

The application is a continuation of application Ser. 5 
No. 07/746,840, filed Aug. 19, 1991 now abandoned. 

BACKGROUND OF THE INVENTION 

1. Field of the Invention 

The present invention relates generally to computer 10 
system user interfaces and more particularly to a com- 
puter system user interface having stereo and filtered 
sound effects for enabling blind or visually impaired 
users to operate a computer system with a graphical 
user interface. 15 

2. Description of the Prior Art 

In recent years, there has been a move among com- 
puter application software developers toward graphical 
user interfaces. In graphical user interfaces, objects are 
presented for users to manipulate in ways that are simi* 20 
lar to the way that they are manipulated in the real 
work place. Objects, such as file cabinets, folders, docu- 
ments, and printers, are displayed on the screen as icons. 
Users manipulate these objects with a mouse to perform 
desired operations. For example, to file a document in a 25 
folder that is located in a file cabinet in the real work 
place, the user opens the file cabinet, locates and opens 
the correct folder, and puts the document inside. In the 
electronic work place of the graphical user interface, 
the user performs a similar process. The user opens the 30 
file cabinet icon, locates the correct folder icon, and 
drops the document icon in the folder. Because this is an 
electronic environment, users do not have to open the 
folder to put the document in it. However, users have 
been able to use their knowledge of a real work place to 35 
perform this operation. 

Normally sighted persons find graphical user inter- 
faces intuitive and easy to work with. However, except 
for an occasional "beep" or "bong", graphical user 
interfaces are virtually silent and the vast majority of 40 
the information they provide to the user is visual. Thus, 
graphical user interfaces are essentially not usable by 
blind or severely visually unpaired people. 

Blind and visually impaired computer users now ben- 
efit from many forms of adaptive technology, including 45 
speech synthesis, large-print processing, braille desktop 
publishing, and voice recognition. However, presently, 
almost none of the foregoing tools is adapted for use 
with graphical user interfaces. It has been suggested 
that programmers could write software with built-in 50 
voice labels for icons. Lazzaro, Windows of Vulnerabil- 
ity. Byte Magazine, June 1991, page 416. Various syn- 
thetic or recorded speech solutions for making com- 
puter display screen contents available to blind persons 
have been suggested, for example in Golding, et. al., 55 
IBM Technical Disclosure Bulletin, Vol. 26, No. 10B, 
pages 5633-5636 (March 1984), and Barnett, et. al., IBM 
Technical Disclosure Bulletin, Vol. 26, No. 10A, pages 
4950-4951 (March 1984). Additionally, there have been 
suggested systems that include a mouse with a braille 60 
transducer so that a blind user may read text and obtain 
certain tactile position feedback from the mouse. 
Comerford, IBM Technical Disclosure Bulletin, Vol, 
28, No. 3, page 1343 (August 1985), AfTmito, et. al., 
IBM Technical Disclosure Bulletin, Vol. 31, No. 12, 65 
page 386 (May 1989). However, while announcing vari- 
ous text items, either audibly or by means of a braille 
transducer in the mouse, may provide some information 



2 

to blind user, it does not enable the user to navigate 
about and locate objects on the computer display 
screen. 

There has been suggested an audible cursor position- 
ing and pixel (picture element) status identification 
mechanism to help a user of an interactive computer 
graphics system locate data by using aural feedback to 
enhance visual feedback. As the cursor is stepped across 
the screen, an audible click is generated that varies in 
tone corresponding in tone to the current status of each 
pixel encountered. With this combination in audible and 
visual cursor feedback, it becomes a simple task to iden- 
tify the desired line by noting the change in tone as the 
cursor moves. For color display applications, each 
color is represented by a distinct tone so any single pixel 
may be distinguished from the surrounding pixels of a 
different color. It has been suggested that this system is 
especially helpful for visually impaired or learning dis- 
abled users. Drumm, et. al., IBM Technical Disclosure 
Bulletin, Vol. 27, No. 48, page 2528 (September 1984). 
However, the foregoing disclosure docs not suggest a 
means of enabling a blind user to navigate about or 
locate objects on the computer display screen. 

SUMMARY OF THE INVENTION 

In the present invention, a stereo balance effect is 
used to convey information about the position of the 
pointer in the left/right or X direction relative to the 
limits of the client area of the current window. The 
system of the present invention includes laterally spaced 
apart audio transducers, which may be speakers or ste- 
reo headphones. As the pointer approaches the left 
boundary of the client area, the sounds representing the 
client area come more and more exclusively from the 
left audio channel. Likewise, approaching the right 
boundary causes the sound to shift to the right channel. 
Centering the pointer within the window causes equal 
sound output from both stereo channels. This audio 
effect is dramatic and effective. It also allows the user to 
sense quickly the size of the window. If the user hears a 
large balance shift for relatively little mouse movement, 
the user can sense that the window is narrow. Addition- 
ally, in the present invention, a different effect is imple- 
mented to communicate relative position in the top/bot- 
tom or Y axis of the window client area. In this aspect 
of the invention, the frequency of the sounds represent- 
ing the client is a function of the top/bottom or Y posi- 
tion of the pointer within the window client area. Pref- 
erably, the frequency is changed in a fixed number of 
discrete steps, which allow the user to count them and 
better ascertain top/bottom or Y position. In the pre- 
ferred embodiment, the frequency is increased as the 
pointer moves from the bottom to the top of the client 
area, which follows the intuitive metaphor of high 
pitched sounds corresponding to a high position. 

A particular problem that blind or visually impaired 
users have in operating graphical user interfaces is navi- 
gating in windows. Windows include a client area that 
is populated with text and or icons. Sighted users can 
find objects within windows at a glance and move the 
pointer to them almost without thinking. However, a 
blind or visually impaired user, even if provided with 
text to speech or other audio identification of the ob- 
jects can find such objects only through trial and error 
or random searching. Moreover, it is very difficult for a 
blind or visually impaired user, after having found and 
identified all of the objects in the window, to navigate 
back to a desired object 
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BRIEF DESCRIPTION OF THE DRAWINGS ^ c L^Zt^^^ 

FIG. 1 is a pictorial view of a window with relative decreases. Similarly, as the user moves pointer 27 

amplitude and frequency scales added to aid in under- toward the right, right channel amplitude increases 

standing the invention. 5 while left channel amplitude deceases. The stereo effect 

FIG. 2 is a block diagram showing a preferred system provided by the present invention enables the user al- 

of the present invention. most to "see" the left/right position of the pointer. 

FIG. 3 is a block diagram showing a preferred imple- As the user moves pointer 27 vertically or in the 

mentation of the sound generator of the present inven- top/bottom axis of window 11, the pitch or frequency 

tioa. 10 of the tone varies in stepwise fashion, as depicted by the 

FIG. 4 is a flowchart of a preferred software imple- scale 31 displayed along the left hand margin of window 

mentation of the present invention. 11. Scale 31 shows graphically the stepwise arrange- 

nFsrRTPTiOK of thf prfffrpfh mcnt of frc< * UCIlcics 83 a function of the top/bottom 

^^^mmt^i^ position of the pointer. In the preferred embodiment, 

mifuuiMtiNi 15 cight distinct frequencies are provided at 300 hertz 

Referring now to the drawings, and first to FIG. 1, a intervals. The stepwise frequency function allows the 

window is designated generally by the numeral 11. user to count the steps and thereby know how close 

Window 11 is displayed on a computer system display pointer 27 is to the top or bottom of window client area 

screen, as is well known to those skilled in the art. Win- 19. The frequency or pitch variation enables the user to 

dow 11 includes a window border 13, a tide bar 15, an 20 visualize accurately the top/bottom position of pointer 

action bar 17, and a client area 19. Title bar 15 includes, 27. Again, scale 31 is illustrated only for ease of explana- 

in addition to the title of the window, a system menu tion and understanding of the invention, it is not actu- 

icon 21, and window-sizing icons 23 and 25. System ally displayed on the screen. 

menu icon 21 allows a user to display a pull-down menu With the present invention, the user can tell easily 
containing the actions that the user can perform on the 25 where pointer 27 is in window client area 19. By con- 
window. Window-sizing icon 23 provides a fast way to vention, title bar 15 and action bar 17 are always located 
use a mouse or other pointing device to minimim the at the top of window 11. The choices in action bar 17 
window, by reducing it to an icon. Conversely, win- are always listed left to right starting near the upper left 
dow-sizing icon 25 provides a fast way for the user to hand corner of window 11. Preferably, the choices of 
maximize the window to fill the entire screen. 30 action bar 17 are announced by text-to-speech or re- 
Action bar 17 contains a list of the actions of an appli- corded speech. Thus, the user can easily find the upper 
cation. The user can cause the system to display a pull- left hand comer of client area 19 and thereby find action 
down menu under each item in action bar 17. bar 17 or system menu icon 21. Similarly, minimize icon 
Client area 19 comprises the remainder of window 11. 23 and maximize icon 25 are always located in the upper 
Client area 19 is the focus of the users attention and it is 35 right hand corner of window 11, which the user can 
where the user is presented with the object or objects find quickly and easily. 

upon which the user wishes to work. As those skilled in Turning now to FIG. 2, there is shown a block dia- 

the art and those familiar windows will recognize, the gram of the system of the present invention. The CPU 

window client area is normally populated with text hardware is contained in dashed rectangle 33. Running 

and/or icons. However, for purposes of clarity and 40 on CPU hardware 33 is an operating system 35 which 

illustration, client area 19 is shown to be empty. includes presentation logic 37. A plurality of applica- 

A pointer 27 is shown within client area 19. Pointer tions 39 are shown running on operating system 35. 

27 is moveable about the screen by means of a mouse Video interface logic and hardware 41 receive inform a- 

(not shown) or other pointing device. The user can tion from presentation logic 37, which is displayed on a 

move pointer 27 to various objects to select, open, or 45 video monitor 43. A mouse 45 and a keyboard 47 pro- 

dixectly manipulate them. People with normal vision vide user input to the system 

can move pointer 27 about the screen and find such The system includes query code 49 which receives 

items as system menu icon 21 or maximize icon 25 eas- information from presentation logic 37 including type 

ily. However, as can be imagined, blind or severely of window, position and size of window, and current 

visually impaired people would have a very difficult 50 pointer position. Query code 49 provides information to 

time locating items in a window. Accordingly, in the sound generation software 51 and hardware 53. The 

present invention, sound effects are provided to give the output from sound generation hardware 53 is provided 

user audible feedback about the position of pointer 27. to stereo headphones 55 or speakers. 

In FIG. 1, a left/right amplitude scale designated Referring now to FIG. 3, there is shown a block 
generally by the numeral 29 is depicted along the bot- 55 diagram of the sound generation software and hardware 
torn margin of window 11. Scale 29 is provided only for of the system of the present invention. Sound genera- 
ease of explanation and understanding of the invention tion hardware 53 includes a white noise generator 57 
and is not actually displayed on the screen. In the pres- and oscillator or oscillators 59. White noise generator 
ent invention, an audible tone is generated from a pair of 57 generates white noise, which sounds like a hiss, 
laterally spaced apart transducers. The transducers may 60 White noise is actually a mixture of different tones or 
be either speakers positioned on opposite sides of the frequencies in the way that white light is a mixture of 
workstation or headphones worn by the user. Scale 29 colored light Oscillators 59 add certain frequency corn- 
shows graphically the relative left/right amplitudes or ponents to the white noise generated by white noise 
balance of the left and right channels as a function of the generator 57 at a summing circuit 61. 
horizontal or left/right position of the pointer. Thus, 65 The sound generation software outputs include a 
when the pointer is positioned on the vertical center line filter center frequency control 63, which operates a 
of client area 19, the amplitudes of the left and right variable bandpass filter 65. Variable bandpass filter 65 
channels are equal to each other and are balanced. As filters out frequency components above and below the 
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filter center frequency and outputs an audio signal hav- 
ing a relatively narrow band of frequencies. The audio 
output of variable bandpass filter 65 is perceived by a 
listener as either a relatively high pitched hiss or rela- 
tively low pitched hiss depending on the filter center 
frequency. 

The output from variable bandpass filter 65 is split at 
67 into left and right channels. A left amplitude control 
69 controls a variable attenuator 71 in the left channel 
and a right amplitude control 73 controls a variable 
attenuator 75 in the right channel. The output from 
variable attenuator 71 is amplified and an output ampli- 
fier 77 and the audio signal is produced at left speaker 
79. Similarly, the output from variable attenuator 75 is 
amplified at an output amplifier 81 and produced as an 
audio signal at right speaker 83. 

Referring now to FIG. 4, there is shown a flowchart 
of a preferred embodiment of the query code of the 
present invention. First, the pointer position (Xptr t 
Yptr) is queried at block 85. Then, at block 87, the 
identity and type of the window indicated by the 
pointer is queried. Then, the system tests at decision 
block 89 whether the window indicated by the pointer 
is of the type that uses stereo and balanced sound ef- 
fects. In the present invention, window is defined 
broadly to include not only application windows as 
described above, but also the background screen, mes- 
sage boxes, dialog boxes, pull-down menus, pop-up 
menus, and the like. In the preferred embodiment of the 
invention, the stereo and balanced sound effects are 
produced only when the pointer is in the client area of 
an application window. Thus, if the pointer is some- 
where other than the client area of an application win- 
dow, the sounds are shut off at block 91 if they are not 
used for some other purpose and the system returns 
again to query pointer position at block 85. 

If the pointer is in the client area of an application 
window, the system queries the windows extents at 
block 93. This amounts to determining the left/right 
limits of the window client area, which are designated 
Xlcft and Xright, respectively, and the top/bottom 
limits of the window client area, which are designated 
Ytop and Ybottom, respectively. Then, at block 95, the 
system calculates the pointer position relative to the 
window extents along the X axis by the formula: 



10 



15 



20 



25 



Fx = 



Xotr - Xlcft 
Xright - Alcft 



Then, at block 97, Px, which is the right channel ampli- 
tude, is output to the right amplitude control and 1-Px, 
which is the left channel amplitude, is output to the left 
amplitude control. Next, at block 99, the system calcu- 
lates the pointer position relative to the window extents 
along the Y axis by the formula: 

- Yotr — ^bottom 
^ " Ytop - ^bottom 

Then, at block 101, the system uses Py to calculate the 
filter center frequency by the formula 300 hertz * 9 
(1 H-int(Py*8)), which is output to the sound generator. 
The formula of block 101 produces a set of stepwise 
frequencies from 300 hertz to 2,400 hertz, as illustrated 
in FIG. 1. After the filter center frequency has been 
output at block 101, the system returns to block 85 and 
again queries pointer position. 

From the foregoing it may be seen that the system of 
the present invention provides a blind or visually im- 



paired user with audio information sufficient to enable 
the user to locate objects in a window. The present 
invention may also find use among normally sighted 
users who desire additional sensory input. 

While the invention has been particularly shown and 
described with reference to a preferred embodiment, 
those skilled in the art will understand that various 
changes may be made in form and detail without depart- 
ing from the spirit and scope of the invention. 
What I claim is: 

1. A method of providing a user of a computer system 
which includes a display screen for displaying at least 
one window on a portion thereof, said at least One 
window including a title bar and a client area, a pointing 
device for manually positioning a pointer on said dis- 
play screen, and a pair of spaced apart speakers, with 
audio information regarding the position of the pointer 
on the display screen which comprises the steps of: 

monitoring the position of the pointer on said display 
screen; and 

generating a distinctive audio signal from each of said 
speakers only when said pointer is positioned in a 
client area of a window, the relative amplitudes of 
said audio signals being proportional to the relative 
left/right position of said pointer in said client area 
of said window. 

2. The method as claimed in claim 1, wherein the 
frequency of said distinctive audio signal is proportional 

30 to the relative top/bottom position of said pointer in 
said client area of said window. 

3. The method as claimed in claim 1, including the 
step of: 

discontinuing said distinctive audio signal whenever 
3 5 said pointer is positioned outside a client area of a 
window. 

4. The method as claimed in claim 1, wherein said 
window includes a left boundary and a right boundary 
of said client area of said window, and said generating 

40 step includes the steps of: 

determining the positions of said left and right bound- 
aries on said display screen; 
calculating the position of said pointer relative to said 
left and right boundaries; and 
45 controlling the amplitude of said distinctive audio 
signal generated by each of said speakers such that 
the relative amplitudes of said audio signals are 
equal to the position of said pointer relative to said 
left and right boundaries. 
50 5. A method of providing a user of a computer system 
which includes a display screen for displaying at least 
one window on a portion thereof, said at least one win- 
dow including a title bar and a client area, and a point- 
ing device for manually positioning a pointer on said 
55 display screen, with audio information regarding the 
position of the pointer on the display screen, which 
comprises the steps of: 
. monitoring the position of the pointer on said display 
screen; and 

60 generating a first distinctive audio signal only when 
said pointer is positioned in a client area of a win- 
dow, wherein the frequency of said first distinctive 
audio signal is proportional to the relative top/bot- 
tom position of said pointer in said client area of 
65 said window. 

6. The method as claimed in claim 5, wherein said 
first distinctive audio emanates from a position located 
to one side of said user and including the step of: 
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generating a second distinctive audio signal emanat- 
ing from, a position located to the opposite side of 
said user, said second distinctive audio signal hav- 
ing a frequency substantially equal to the frequency 
of said first audio signal. 

7. The method as claimed in claim 6, wherein the 
relative amplitudes of said first and second distinctive 
audio signals are proportional to the relative right/left 
position of said pointer in said client area of said win- 
dow. 

8. The method as claimed in claim 7, wherein said 
first distinctive audio signal emanates from a right 
speaker and said second distinctive audio signal ema- 
nates from a left speaker, and the amplitude of said first 



10 



8 



distinctive audio signal increases as said pointers move 
toward the right in said client area of said window and 
the amplitude of said second distinctive audio signal 
increases as said pointers move toward the left in said 
client area of said window. 

9. The method as claimed in claim 5, wherein said 
frequency increases in stepwise fashion as said pointer is 
moved toward the top of said screen. 

10. The method as claimed in claim 5, including the 
step of: 

discontinuing said distinctive audio signal whenever 
said pointer is positioned outside said client area of 
said window. 



IS 



20 



25 



30 



35 



40 



45 



50 



55 



60 



65 



03/05/2004, EAST Version: 1.4.1 



